error analysis lecture 10

7/30/2019 Error analysis lecture 10

1/32

Physics 509 1

Physics 509: Error Propagation, and

the Meaning of Error Bars

Scott OserLecture #10

October 14, 2008


2/32

Physics 509 2

What is an error bar?Someone hands you a

plot like this. What dothe error bars indicate?

Answer: you can never

be sure, unless it'sspecified!

Most common: verticalerror bars indicate 1uncertainties.

Horizontal error bars canindicate uncertainty onX coordinate, or can

indicate binning.

Correlations unknown!


3/32

Physics 509 3

Relation of an error bar to PDF shape

The error bar on a plot ismost often meant torepresent the 1 uncertaintyon a data point. Bayesiansand frequentists will disagree

on what that means.

If data is distributed normallyaround true value, it's clear

what is intended:

exp[-(x-)2/22].

But for asymmetric

distributions, different thingsare sometimes meant ...


4/32

Physics 509 4

An error bar is a shorthand approximation to aPDF!

In an ideal Bayesian universe, error bars don't exist.Instead, everyone will use the full prior PDF and thedata to calculate the posterior PDF, and then report

the shape of that PDF (preferably as a graph ortable).

An error bar is really a shorthand way to parameterize

a PDF. Most often this means pretending the PDF isGaussian and reporting its mean and RMS.

Many sins with error bars come from assumingGaussian distributions when there aren't any.


5/32

Physics 509 5

An error bar as a confidence interval

Frequentist techniques don't directly answer the question of what theprobability is for a parameter to have a particular value. All you cancalculate is the probability of observing your data given a value of theparameter.The confidence interval construction is a dodge to get

around this. Starting point is thePDF for the estimator,for a fixed value of theparameter.

The estimator hasprobability 1 tofall in the white region.


6/32

Physics 509 6

Confidence interval construction

The confidence band isconstructed so that the averageprobability of

truelying in the

confidence interval is 1-.

This should not be interpretedto mean that there is, forexample, a 90% chance thatthe true value of is in theinterval (a,b) on any given trial.

Rather, it means that if you ranthe experiment 1000 times, andproduced 1000 confidenceintervals, then in 90% of these

hypothetical experiments (a,b)would contain the true value.

Obviously a very roundaboutway of speaking ...


7/32

Physics 509 7

The ln(L) rule

It is not trivial to construct properfrequentist confidence intervals.Most often an approximation isused: the confidence interval for asingle parameter is defined as therange in which ln(L

max)-ln(L)


8/32

Physics 509 8

Error-weighted averages

Suppose you have N independent measurements of a quantity.You average them. The proper error-weighted average is:

x = x i / i

2

1/i

2

Vx =1

1 / i2

If all of the uncertainties are equal, then this reduces to the simplearithmetic mean, with V() = V(x)/N.


9/32

Physics 509 9

Bayesian derivation of error-weightedaverages

Suppose you have N independent measurements of a quantity,distributed around the true value with Gaussian distributions.For flat prior on we get:

Vx =1

1 / i2

Pxi exp [12

x i i 2

]=exp

[12i

x i i 2

]It's easy to see that this has the form of a Gaussian. To find itspeak, set derivative with respect to equal to zero.

dP

d=exp [12i x i i

2

] [i x i i2 ]=0 =xi /i

2

1 / i2

Calculating the coefficient of2 in the exponent yields:


10/32

Physics 509 10

Averaging correlated measurements

We already saw how to average N independent measurement.What if there are correlations among measurements?

For the case of uncorrelated Gaussianly distributedmeasurements, finding the best fit value was equivalent tominimizing the chi-squared:

2=i x i i

2

In Bayesian language, this comes about because the PDF for isexp(-2/2). Because we know that this PDF must be Gaussian:

Pexp

[1

2

0

2

]then an easy way to find the 1 uncertainties on is to find the

values of for which 2 = 2min

+ 1.


11/32

Physics 509 11

Averaging correlated measurements II

The obvious generalization for correlated uncertainties is to formthe 2 including the covariance matrix:

2=

i

j

x ix jV1ij

We find the best value of by minimizing this 2 and can then findthe 1 uncertainties on by finding the values of for which2 = 2

min+ 1.

This is really parameter estimation with one variable.

The best-fit value is easy enough to find:

=

i , j

x j V1ij

i , j

V1ij


12/32

Physics 509 12

Averaging correlated measurements III

Recognizing that the 2 really just is the argument of anexponential defining a Gaussian PDF for ...

2=

i

j

x ix jV1ij

we can in fact read off the coefficient of2, which will be 1/V():

2=

1

i , j V1

ij

In general this can only be computed by inverting the matrix asfar as I know.


13/32

Physics 509 13

Averaging correlated measurements IVSuppose that we have N

correlated measurements. Eachhas some independent error=1 and a common error b thatraises or lowers them alltogether. (You would simulateby first picking a random valuefor b, then for eachmeasurement picking a newrandom value c with RMS and

writing out b+c.

Each curve shows how the erroron the average changes with N,for different values of b.

b=1b=0.5b=0.25b=0


14/32

Physics 509 14

Averaging correlated measurements: exampleConsider the following example, adapted from Cowan's book:

We measure an object's length with two rulers. Both are calibrated to beaccurate at T=T

0, but otherwise have a temperature dependency: true length y is

related to measured length by:

We assume that we know the ciand the uncertainties, which are Gaussian. We

measure L1, L

2, and T, and so calculate the object's true length y.

y i=Lici TT0

y i=Lici TT0

We wish to combine the measurements from the two rulers to get our best estimate of

the true length of the object.


15/32

Physics 509 15

Averaging correlated measurements: exampleWe start by forming the covariance matrix of the two measurements:

y i=Lici TT0 i2=L

2c i

2T

2

We use the method previously described to calculate the weighted average for thefollowing parameters:

c1

= 0.1 L1=2.0 0.1 y

1=1.80 0.22 T

0=25

c2

= 0.2 L2=2.3 0.1 y

2=1.90 0.41 T = 23 2

Using the error propagation equations, we get for the weighted average:

ytrue

= 1.75 0.19

WEIRD: the weighted average is smaller than either measurement! What's going

on??

cov y1, y 2=c1 c2T2


16/32

Physics 509 16

Averaging correlated measurements: example

Bottom line:

Because y1

and y2

disagree, fit

attempts to adjusttemperature tomake them agree.This pushes fittedlength lower.

This is one of manycases in which thedata itself gives

you additionalconstraints on thevalue of asystematic (here,what is the true T).


17/32

Physics 509 17

The error propagation equation

Let f(x,y) be a function of two variables, and assume that theuncertainties onxand yare known and small. Then:

f2= dfdx

2

x2 dfdy

2

y22 dfdx

df

dy x yThe assumptions underlying the error propagation equation are:

covariances are known fis an approximately linear function ofxand yover the span ofxdxorydy.

The most common mistake in the world: ignoring the third term.

Intro courses ignore its existence entirely!


18/32

Physics 509 18

Example: interpolating a straight line fit

Straight line fit y=mx+b

Reported values from astandard fitting package:

m = 0.658 0.056b = 6.81 2.57

Estimate the value and

uncertainty ofywhenx=45.5:

y=0.658*45.5+6.81=36.75

UGH! NONSENSE!

dy=2.57245.5.0562=3.62


19/32

Physics 509 19

Example: straight line fit, done correctly

Here's the correct way to estimate y at x=45.5. First, I find a betterfitter, which reports the actual covariance matrix of the fit:

m = 0.0658 + .056b = 6.81 + 2.57

= -0.9981

dy=2.5720.05645.522 0.99810.05645.52.57=0.16

(Since the uncertainty on each individual data point was 0.5, and thefitting procedure effectively averages out their fluctuations, then weexpect that we could predict the value of y in the meat of thedistribution to better than 0.5.)

Food for thought: if the correlations matter so much, why don't mostfitting programs report them routinely???


20/32

Physics 509 20

Reducing correlations in the straight line fit

The strong correlationbetween m and b resultsfrom the long lever arm---since you must extrapolateline to x=0 to determine b, a

big error on m makes a bigerror on b.

You can avoid strongcorrelations by using more

sensible parameterizations:for example, fit data toy=b'+m(x-45.5):

b' = 36.77 0.16

m = 0.658 .085 = 0.43

dy at x=45.5 = 0.16


21/32

Physics 509 21

Non-Gaussian errors

The error propagation can give the false impression thatpropagating errors is as simple as plugging in variances andcovariances into the error propagation equation and thencalculating an error on output.

However, a significant issue arises: although the error propagationequation is correct as far as it goes (small errors, linearapproximations, etc), it is often not true that the resultinguncertainty has a Gaussian distribution! Reporting the central

value and an RMS may be misleading.


22/32

Physics 509 22

Ratio of two Gaussians I

Consider the ratio R=x/y of two independent variables drawn fromnormal distributions. By the error propagation equation

dRR 2

= dxx 2

dyy 2

Let's suppose x = 1 0.5 and y = 5 1. Then the calculated valueof R = 0.200 .108.

What does the actual distribution for R look like?


23/32

Physics 509 23

Ratio of two Gaussians II

x = 1 0.5 and y = 5 1.

Error propagation prediction:R = 0.200 .108.

Mean and RMS of R:0.208 0.118

Gaussian fit to peak:0.202 0.107

Non-Gaussian tails evident,

especially towards larger R,but not too terrible.


24/32

Physics 509 24

Ratio of two Gaussians III

x = 5 1 and y = 1 0.5

Error propagation:R = 5.00 2.69.



Completely non-Gaussian in

all respects. Occasionallywe even divide by zero, orclose to it!


25/32

Physics 509 25

Ratio of two Gaussians IV

x = 5 1 and y = 5 1

Error propagation:R = 1 0.28



More non-Gaussian than first case,much better than second.

Rule of thumb: ratio of two

Gaussians will be approximatelyGaussian if fractional uncertaintyis dominated by numerator, anddenominator cannot be smallcompared to numerator.


26/32

Physics 509 26

Ratio of two Gaussians: testing an asymmetry

A word of caution: often scientists like to form asymmetries---forexample, does the measurement from the left half of the apparatusagree with that from the right half? Asymmetry ratios are the usualway to do this, since common errors will cancel in ratio:

A= LR1

2LR

Be extremely careful about using the error on the asymmetry as a

measure of whether A is consistent with zero. In other words,A=0.5 0.25 is usually not a 2 sigma result, probability-wise.

Instead, it's better to simply test whether the numerator is consistent

with zero or not.

G li i h i i


27/32

Physics 509 27

Generalizing the error propagation equation

cov fk , fl=i

j

fkx i

flx j

cov x i, x j

If we have N functions of M variables, we can calculate theircovariance by:

We can write this compactly in matrix form as:

G=Gki

=

fk

xi

(an N X M matrix)

Vf=

G

Vx

G

T

A t i


28/32

Physics 509 28

Asymmetric errors

The error propagation equation works with covariances. But the

confidence interval interpretation of error bars often reportsasymmetric errors. We can't use the error propagation equation tocombine asymmetric errors. What do we do?

Quite honestly, the typical physicist doesn't have a clue. The mostcommon procedure is to separately add the negative error barsand the positive error bars in quadrature:

Source - Error + Error

Error A -0.025 +0.050

Error B -0.015 +0.010

Error C -0.040 +0.040

Combined -0.049 +0.065

Warning: in spite of how commonthis procedure is and what youmay have heard, it has nostatistical justification and often

gives the wrong answer!

H t h dl t i


29/32

Physics 509 29

How to handle asymmetric errors

Best case: you know the likelihood function (at least numerically). Report

it, and include it in your fits.

More typical: all you know are a central value, with +/- errors. Your onlyhope is to come up with some parameterization of the likelihood that

works well. This is a black art, and there is no generally satisfactorysolution.

Barlow recommends (in a paper on his web site):

lnL =12

2

1212

You can verify that this expression evaluates to when =+1

or=2

Parameterizing the asymmetric likelihood


30/32

Physics 509 30

a a ete g t e asy et c e oodfunction

lnL =

1

2

2

1212

Note that parameterizationbreaks down when

denominator becomesnegative. It's like sayingthere is a hard limit on atsome point. Use this onlyover its appropriate range ofvalidity.

Barlow recommends this formbecause it worked well for avariety of sample problemshe tried---it's completelyempirical.

See Barlow, Asymmetric

Statistical Errors,arXiv:physics/0406120

Application: adding two measurements with


31/32

Physics 509 31

pp gasymmetric errors.

ln L A , B=1

2

A52

1212A5

1

2

B32

1313B3

Suppose A=521

and B=331

. What is the confidence interval for A+B?

Let C=A+B. Rewrite likelihood to be a function of C and one ofthe other variables---eg.

ln L C , A=1

2

A52

1212A5

1

2

CA32

1313CA3

To get the likelihood function for C alone, which is what we care

about, minimize with respect to nuisance parameter A:

lnL C=minA

ln L C , A

Asymmetric error results


32/32

Physics 509 32

Asymmetric error results

error analysis lecture 10

Documents