measurement uncertainties and inconsistencies dr. richard young optronic laboratories, inc

Optronic Laboratories, Inc.Optronic Laboratories, Inc.

Measurement Measurement Uncertainties and Uncertainties and InconsistenciesInconsistencies

Measurement Measurement Uncertainties and Uncertainties and InconsistenciesInconsistencies

Dr. Richard YoungDr. Richard Young


Dr. Richard YoungDr. Richard Young



IntroductionIntroductionIntroductionIntroduction

The concept of accuracy is generally The concept of accuracy is generally understood.understood.

“…“…an accuracy of 1%.”an accuracy of 1%.” What does this mean?What does this mean?

•99% inaccurate?99% inaccurate?

The concept of accuracy is generally The concept of accuracy is generally understood.understood.

“…“…an accuracy of 1%.”an accuracy of 1%.” What does this mean?What does this mean?

•99% inaccurate?99% inaccurate?



The confusion between the concept The confusion between the concept and the numbers has lead national and the numbers has lead national laboratories to abandon the term laboratories to abandon the term accuracy.accuracy. Except in qualitative terms e.g. Except in qualitative terms e.g.

high accuracy.high accuracy.The term now used is uncertainty.The term now used is uncertainty.

“…“…an uncertainty of 1%.”an uncertainty of 1%.”

The confusion between the concept The confusion between the concept and the numbers has lead national and the numbers has lead national laboratories to abandon the term laboratories to abandon the term accuracy.accuracy. Except in qualitative terms e.g. Except in qualitative terms e.g.

high accuracy.high accuracy.The term now used is uncertainty.The term now used is uncertainty.

“…“…an uncertainty of 1%.”an uncertainty of 1%.”



Sometimes…Sometimes… Users do not know the Users do not know the

uncertainty of their results.uncertainty of their results. They interpret any variations as They interpret any variations as

inconsistencies.inconsistencies.

Sometimes…Sometimes… Users do not know the Users do not know the

uncertainty of their results.uncertainty of their results. They interpret any variations as They interpret any variations as

inconsistencies.inconsistencies.


Uncertainty vs. Uncertainty vs. InconsistencyInconsistency

Uncertainty vs. Uncertainty vs. InconsistencyInconsistency

Laboratories give different values, Laboratories give different values, but the difference is within their but the difference is within their combined uncertainties…combined uncertainties… Pure chance.Pure chance.

Laboratories give different values, Laboratories give different values, and the difference is outside their and the difference is outside their combined uncertainties…combined uncertainties… Inconsistency.Inconsistency.

Laboratories give different values, Laboratories give different values, but the difference is within their but the difference is within their combined uncertainties…combined uncertainties… Pure chance.Pure chance.

Laboratories give different values, Laboratories give different values, and the difference is outside their and the difference is outside their combined uncertainties…combined uncertainties… Inconsistency.Inconsistency.


What is uncertainty?What is uncertainty?What is uncertainty?What is uncertainty?

“…“…an uncertainty of 1%.”an uncertainty of 1%.” But is 1% the maximum, average But is 1% the maximum, average

or typical variation users can or typical variation users can expect?expect?

Uncertainty is a statistical quantity Uncertainty is a statistical quantity based on the average and standard based on the average and standard deviation of data.deviation of data.

“…“…an uncertainty of 1%.”an uncertainty of 1%.” But is 1% the maximum, average But is 1% the maximum, average

or typical variation users can or typical variation users can expect?expect?

Uncertainty is a statistical quantity Uncertainty is a statistical quantity based on the average and standard based on the average and standard deviation of data.deviation of data.


StatisticsStatisticsStatisticsStatistics

““There are three types of lies: There are three types of lies: lies, damned lies and statistics.lies, damned lies and statistics.””

--attributed to Benjamin Disraeliattributed to Benjamin Disraeli

““There are three types of lies: There are three types of lies: lies, damned lies and statistics.lies, damned lies and statistics.””

--attributed to Benjamin Disraeliattributed to Benjamin Disraeli

““The difference between statistics and The difference between statistics and experience is time.”experience is time.”

--Richard YoungRichard Young

““The difference between statistics and The difference between statistics and experience is time.”experience is time.”

--Richard YoungRichard Young

Statistics uses past experience to predict Statistics uses past experience to predict likely future events.likely future events.

Statistics uses past experience to predict Statistics uses past experience to predict likely future events.likely future events.


StatisticsStatisticsStatisticsStatistics

We toss a coin:We toss a coin: It is equally likely to be heads or It is equally likely to be heads or

tails.tails.We toss two coins at the same time:We toss two coins at the same time:

There are 4 possible outcomes:There are 4 possible outcomes:• Head + Head• Head + Tail• Tail + Head• Tail + Tail

We toss a coin:We toss a coin: It is equally likely to be heads or It is equally likely to be heads or

tails.tails.We toss two coins at the same time:We toss two coins at the same time:

There are 4 possible outcomes:There are 4 possible outcomes:• Head + Head• Head + Tail• Tail + Head• Tail + Tail

These 2 are the same These 2 are the same and hence twice as and hence twice as

likely to happen as the likely to happen as the others.others.

These 2 are the same These 2 are the same and hence twice as and hence twice as

likely to happen as the likely to happen as the others.others.


StatisticsStatisticsStatisticsStatistics Now let us throw Now let us throw

10 coins.10 coins. There are 1024 There are 1024

possibilities (2possibilities (21010).). What if we threw What if we threw

them 1024 times, them 1024 times, and counted each and counted each time a certain time a certain number of heads number of heads resulted…resulted…

Now let us throw Now let us throw 10 coins.10 coins.

There are 1024 There are 1024 possibilities (2possibilities (21010).).

What if we threw What if we threw them 1024 times, them 1024 times, and counted each and counted each time a certain time a certain number of heads number of heads resulted…resulted…

0

50

100

150

200

250

300

0 1 2 3 4 5 6 7 8 9 10

Number of Heads

Nu

mb

er o

f O

ccu

rren

ces


StatisticsStatisticsStatisticsStatisticsAlthough the Although the

outcome of each outcome of each toss is random…toss is random…

...not every result ...not every result is equally likely.is equally likely.

If we divide the If we divide the number of number of occurrences by the occurrences by the total number of total number of throws…throws… We get We get probability.probability.

Although the Although the outcome of each outcome of each toss is random…toss is random…

...not every result ...not every result is equally likely.is equally likely.

If we divide the If we divide the number of number of occurrences by the occurrences by the total number of total number of throws…throws… We get We get probability.probability.

0

50

100

150

200

250

300

0 1 2 3 4 5 6 7 8 9 10

Number of Heads

Nu

mb

er o

f O

ccu

rren

ces


StatisticsStatisticsStatisticsStatisticsHere is the same Here is the same

plot, but shown as plot, but shown as probability.probability.

Probability is just a Probability is just a number that number that describes the describes the likelihood between:likelihood between: 0 = never 0 = never happenshappens

1 = always 1 = always happenshappens

Here is the same Here is the same plot, but shown as plot, but shown as probability.probability.

Probability is just a Probability is just a number that number that describes the describes the likelihood between:likelihood between: 0 = never 0 = never happenshappens

1 = always 1 = always happenshappens

0

0.05

0.1

0.15

0.2

0.25

0.3

0 1 2 3 4 5 6 7 8 9 10

Number of Heads

Pro

bab

ility

of

Occ

urr

ence


StatisticsStatisticsStatisticsStatisticsGauss described a Gauss described a

formula that formula that predicted the predicted the shape of any shape of any distribution of distribution of random events.random events. Shown in redShown in red

It uses just 2 It uses just 2 values:values: The averageThe average The standard The standard deviationdeviation

Gauss described a Gauss described a formula that formula that predicted the predicted the shape of any shape of any distribution of distribution of random events.random events. Shown in redShown in red

It uses just 2 It uses just 2 values:values: The averageThe average The standard The standard deviationdeviation

0

0.05

0.1

0.15

0.2

0.25

0.3

0 1 2 3 4 5 6 7 8 9 10

Number of Heads

Pro

bab

ility

of

Occ

urr

ence


StatisticsStatisticsStatisticsStatistics Now throw 100 coins…Now throw 100 coins… Now throw 100 coins…Now throw 100 coins…

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 10 20 30 40 50 60 70 80 90 100

Number of Heads

Pro

bab

ility

of

Occ

ure

nce

We have an average We have an average = 50= 50

We have an average We have an average = 50= 50

And a standard And a standard deviation = 5deviation = 5

And a standard And a standard deviation = 5deviation = 5

And the familiar And the familiar bell-shaped bell-shaped distribution.distribution.

And the familiar And the familiar bell-shaped bell-shaped distribution.distribution.

The The Gaussian Gaussian curve fits curve fits exactly.exactly.

The The Gaussian Gaussian curve fits curve fits exactly.exactly.


ConfidenceConfidenceConfidenceConfidence Now throw 100 coins…Now throw 100 coins… Now throw 100 coins…Now throw 100 coins…

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 10 20 30 40 50 60 70 80 90 100

Number of Heads

Pro

bab

ility

of

Occ

ure

nce

Since the total Since the total probability must =1, probability must =1,

the standard the standard deviation marks off deviation marks off certain probabilities.certain probabilities.




0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 10 20 30 40 50 60 70 80 90 100

Number of Heads

Pro

bab

ility

of

Occ

ure

nce






About 67% of About 67% of all results lie all results lie

within within 1 1 standard standard deviation.deviation.


within within 1 1 standard standard deviation.deviation.

““I am 67% confident that a I am 67% confident that a new throw will give new throw will give

between 45 and 55 heads.”between 45 and 55 heads.”




0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0 10 20 30 40 50 60 70 80 90 100

Number of Heads

Pro

bab

ilit

y o

f O

ccu

ren

ce







within within 2 2 standard standard

deviations.deviations.


within within 2 2 standard standard

deviations.deviations.






Real DataReal DataReal DataReal Data

Real data, such as the result of a Real data, such as the result of a measurement, is also characterized measurement, is also characterized by an average and standard by an average and standard deviation.deviation.

To determine these values, we must To determine these values, we must make measurements.make measurements.

Real data, such as the result of a Real data, such as the result of a measurement, is also characterized measurement, is also characterized by an average and standard by an average and standard deviation.deviation.

To determine these values, we must To determine these values, we must make measurements.make measurements.


Real DataReal DataReal DataReal Data NVIS radiance measurements are unusual.NVIS radiance measurements are unusual.

The signal levels at longer wavelengths The signal levels at longer wavelengths can be very low – close to the dark can be very low – close to the dark level of the system.level of the system.

The signal levels at longer wavelengths The signal levels at longer wavelengths dominate the NVIS radiance result.dominate the NVIS radiance result.

The uncertainty in results close to the The uncertainty in results close to the dark level can be dominated by PMT dark level can be dominated by PMT noise.noise.

Therefore: Variations in NVIS results can Therefore: Variations in NVIS results can be dominated by PMT noise.be dominated by PMT noise.

NVIS radiance measurements are unusual.NVIS radiance measurements are unusual. The signal levels at longer wavelengths The signal levels at longer wavelengths

can be very low – close to the dark can be very low – close to the dark level of the system.level of the system.

The signal levels at longer wavelengths The signal levels at longer wavelengths dominate the NVIS radiance result.dominate the NVIS radiance result.

The uncertainty in results close to the The uncertainty in results close to the dark level can be dominated by PMT dark level can be dominated by PMT noise.noise.

Therefore: Variations in NVIS results can Therefore: Variations in NVIS results can be dominated by PMT noise.be dominated by PMT noise.



The net signal from the PMT is used The net signal from the PMT is used to calculate the spectral radiance.to calculate the spectral radiance.

Dark current, which is subtracted Dark current, which is subtracted from each current reading during a from each current reading during a scan, contains PMT noise.scan, contains PMT noise.

Scans at low signals contain PMT Scans at low signals contain PMT noise.noise.

The net signal from the PMT is used The net signal from the PMT is used to calculate the spectral radiance.to calculate the spectral radiance.

Dark current, which is subtracted Dark current, which is subtracted from each current reading during a from each current reading during a scan, contains PMT noise.scan, contains PMT noise.

Scans at low signals contain PMT Scans at low signals contain PMT noise.noise.



PMT noise present in each of these PMT noise present in each of these current readings does not have the current readings does not have the same effect on results:same effect on results: A high or low dark reading will A high or low dark reading will

raise or lower ALL points.raise or lower ALL points. Current readings during scans Current readings during scans

contain highs and lows that contain highs and lows that cancel out to some degree.cancel out to some degree.

PMT noise present in each of these PMT noise present in each of these current readings does not have the current readings does not have the same effect on results:same effect on results: A high or low dark reading will A high or low dark reading will

raise or lower ALL points.raise or lower ALL points. Current readings during scans Current readings during scans

contain highs and lows that contain highs and lows that cancel out to some degree.cancel out to some degree.



1.7E-12

1.8E-12

1.9E-12

2E-12

2.1E-12

2.2E-12

2.3E-12

0 20 40 60 80 100 120 140 160 180 200

Measurement #

Dar

k C

urr

ent

[A]

Excel: “= average()” Excel: “= average()” 2E-12 2E-12Excel: “= average()” Excel: “= average()” 2E-12 2E-12Excel: “= stdev()” Excel: “= stdev()” 1E-13 1E-13Excel: “= stdev()” Excel: “= stdev()” 1E-13 1E-13



-6E-13

-4E-13

-2E-13

0

2E-13

4E-13

6E-13

0 20 40 60 80 100 120 140 160 180 200

Measurement #

Ne

t s

ign

al

[A]

Dark = min



-6E-13

-4E-13

-2E-13

0

2E-13

4E-13

6E-13

0 20 40 60 80 100 120 140 160 180 200

Measurement #

Ne

t s

ign

al

[A]

Dark = min

Dark = max



-6E-13

-4E-13

-2E-13

0

2E-13

4E-13

6E-13

0 20 40 60 80 100 120 140 160 180 200

Measurement #

Ne

t s

ign

al

[A]

Dark = minDark = maxDark = average


dGC

dGAs

A

NVISa

930

450930

450

11

CalculationsCalculationsCalculationsCalculations We can describe the effects of noise on We can describe the effects of noise on

class A NVIS radiance mathematically:class A NVIS radiance mathematically: s s is the standard deviation of the noiseis the standard deviation of the noise C(C() is the calibration factors) is the calibration factors GGAA(() is the relative response of class A ) is the relative response of class A

NVISNVIS

We can describe the effects of noise on We can describe the effects of noise on class A NVIS radiance mathematically:class A NVIS radiance mathematically: s s is the standard deviation of the noiseis the standard deviation of the noise C(C() is the calibration factors) is the calibration factors GGAA(() is the relative response of class A ) is the relative response of class A

NVISNVIS

Dark subtractionDark subtractionDark subtractionDark subtraction

Signal averagingSignal averagingSignal averagingSignal averaging


CalculationsCalculationsCalculationsCalculations

A similar equation, but using NVIS A similar equation, but using NVIS class B response instead of class A, class B response instead of class A, can give the standard deviation in can give the standard deviation in NVISb radiance.NVISb radiance.

The standard deviations should be The standard deviations should be scaled to the luminance to give the scaled to the luminance to give the expected variations in scaled NVIS expected variations in scaled NVIS radiance.radiance.

A similar equation, but using NVIS A similar equation, but using NVIS class B response instead of class A, class B response instead of class A, can give the standard deviation in can give the standard deviation in NVISb radiance.NVISb radiance.

The standard deviations should be The standard deviations should be scaled to the luminance to give the scaled to the luminance to give the expected variations in scaled NVIS expected variations in scaled NVIS radiance.radiance.


CalculationsCalculationsCalculationsCalculations

Noise can be reduced by multiple Noise can be reduced by multiple measurements.measurements.

If we generalize the equation to If we generalize the equation to include multiple dark readings (Ninclude multiple dark readings (NDD) )

and scans (S):and scans (S):

Noise can be reduced by multiple Noise can be reduced by multiple measurements.measurements.

If we generalize the equation to If we generalize the equation to include multiple dark readings (Ninclude multiple dark readings (NDD) )

and scans (S):and scans (S):

930

450930

450

930

450

dGC

dGNS

dGN

As

AD

AD

NVISa Brain overloadBrain overloadBrain overloadBrain overload


SpreadsheetSpreadsheetSpreadsheetSpreadsheet

Moving on to the benefits…Moving on to the benefits…Moving on to the benefits…Moving on to the benefits…

IntroducingIntroducingIntroducingIntroducing

The SpreadsheetThe SpreadsheetThe SpreadsheetThe Spreadsheet

measurement uncertainties and inconsistencies dr. richard young optronic laboratories, inc

Documents