everything you always wanted to know about limits*
DESCRIPTION
Everything You Always Wanted To Know About Limits*. Roger Barlow Manchester University YETI06. *But were afraid to ask. Summary. Prediction confronts data & sees small/zero signal. Frequentist probability and Confidence Level language. Likelihood. Bayesian Probability (Health - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/1.jpg)
Everything You Always Wanted To Know About
Limits*Roger Barlow
Manchester UniversityYETI06
*But were afraid to ask
![Page 2: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/2.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 2
SummaryPrediction
confronts data & sees
small/zero signal
Frequentist
probability and
Confidence
Level languageBayesian
Probability
(Health
Warning)Gaussian
ln L= -½
Zero events
Few events:
Confidence belt The horrendous
case of large
backgrounds
Extension to
several
parameters
Likelihood
![Page 3: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/3.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 3
Model predictionsInput model and parameters
Low energyLagrangian
Feynman Rules for Feynman
diagrams
Cross Sectionsand Branching
RatiosExperiment duration,
luminosity,Efficiency etc
Number of events
Monte Carloprograms
Cuts designed tobring out signal
Data
![Page 4: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/4.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 4
What happens if there’s nothing there?
Even if your analysis finds no events, this is still useful information about the way the universe is built
Want to say more than: “We looked for X, we didn’t see it.”
Need statistics – which can’t prove anything.
“We show that X probably has a mass greater than../a coupling smaller than…”
![Page 5: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/5.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 5
Probability(1): Frequentist
Define Probability of X as P(X)=Limit N∞ N(X)/N
Examples: coins, dice, cards For continuous x extend to Probability
DensityP(x to x+dx)=p(x)dx
Examples: • Measuring continuous quantities (p(x)
often Gaussian)• Parton momentum fractions (proton pdfs)
![Page 6: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/6.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 6
Digression: likelihood
Probability distribution of random variable x often depends on some parameter a.
Joint function p(x,a)Considered as p(x)|a this is the pdf.
Normalised: ∫p(x)dx=1Considered as p(a)|x this is the Likelihood L(a)Not ‘likelihood of a’ but ‘likelihood that a
would give x’Not normalised. Indeed, must never be
integrated.
![Page 7: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/7.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 7
Limitation of Frequentist Probability
Have to say“The statement ‘It will rain tomorrow.’ is
probably true.”Can then even quantify (meteorology).
Can’t say“It will probably rain tomorrow.”
There is only one tomorrow. P is either 1 or 0
![Page 8: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/8.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 8
Interpreting physics results: Mt =173±2 GeV/c2
Can’t say ‘Mt has a 68% probability of lying between 171
and 175 GeV/c2’Have to say‘The statement “Mt lies between 171 and 175
GeV/c2”has a 68% probability of being true’i.e. if you always say a value lies within its error
bars, you will be right 68% of the time.Say “Mt lies between 171 and 175 GeV/c2” with
68% Confidence. Or 169-177 with 95% confidence. Or…
![Page 9: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/9.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 9
Interpreting null resultYour analysis searches for events. Sees none.Use Poisson formula: P(n; )=e-n/n!Small could well give 0 events =0.5 gives P(0)=61% =1.0 gives P(0)=37% =2.3 gives P(0)=10% =3.0 gives P(0)=5%If you always say ‘ 3.0’ you will be right (at least)
95% of the time. 3.0 – with 95% confidence (a.k.a 5% significance.)
‘If is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less.’
given by model parameters. Limit on translates to limit on mass, coupling, ,branching ratio or whatever
![Page 10: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/10.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 10
Probability(2): Bayesian
P(X) expresses by degree of belief in XCan calibrate against cards, dice, etc.Extend to probability density p(x) as
beforeNo restrictions on X or x. Rain, MT, MH,
whateverInterpret physics results using Bayes’
Theorem:pposterior(a|data) p(data|a) x pprior(a)
![Page 11: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/11.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 11
Bayes at work
= x
P(0 events|)
(Likelihood)
Prior: uniformPosterior P()
3 P() d= 0.95
0
Same as Frequentist limit - Happy coincidence
Zero events seen
![Page 12: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/12.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 12
Bayes at work again
= x
P(0 events|) Prior: uniform in ln Posterior P()
3 P() d >> 0.95
0
Is that uniform prior really credible?
Upper limit totally different!
![Page 13: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/13.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 13
Bayes: the bad news• The prior affects the posterior. It is your choice. That
makes the measurement subjective. This is BAD. (We’re physicists, dammit!)
• A Uniform Prior does not get you out of this.• SUSY ‘parameter space’ is not a ‘phase space’• Attempts to invent universally-agreed priors
(‘Objective’ and/or ‘Reference’ Priors) have not worked
Better news: If there is a lot of data then the prejudicial effects of the choice of prior can be small.
• This should ALWAYS be checked for (‘robustness under choice of prior’.)
![Page 14: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/14.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 14
Frequentist versus Bayesian?
Statisticians do a lot of work with Bayesian statistics and there are a lot of useful ideas. But they are careful about checking for robustness under choice of prior.
Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness.
Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness.) But ALWAYS be aware which you are using.
![Page 15: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/15.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 15
A Gaussian MeasurementNo problems
p(x)=exp[-(x-)2/2 2]/√2x: symmetric
x is within ± of with 68% probability is within ± of x at 68% confidencex is above -1.645 with 95% probability is below x+1.645 at 95% confidenceChoice of confidence level and arrangement
Can read regions off log likelihood plot asL(a)=exp[-(x-)2/2 2]/√2
Ln L -(-x)2/2 2
68% region corresponds to fall of ½ from peak
![Page 16: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/16.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 16
A Poisson measurement
You detect 5 events. Best value 5. But what about the errors?
1. 5±√5=5±2.24 Assumes e-n/n! is Gaussian in n. True only for large - and 5 is small
2. Find points where log likelihood falls by ½.
Assumes e-n/n! is Gaussian in .Gives upper error of 2.58, lower error of 1.92
![Page 17: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/17.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 17
3: Doing it properly: Confidence belt (Neyman
interval)Use e-n/n! For any true the
probability that (n, ) is within the belt is 68% (or more) by construction
For any n, lies in [-, +] at 68% confidence
Get upper error 3.38, lower error 2.16
n
-
+
68%
16%16%
Technique works for any CL, and single or double sided
![Page 18: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/18.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 18
Consumer guide
ln L =- ½ is a standard and easy to use. Fine for everyday use. (Though for a simple count the Neyman limit is quite easy)
For 90% 1-sided (upper) limit use ln L =-0.82 (1.28 ) For 95% use ln L =-1.35 (1.645 ) Just plot the likelihood and read off the
value. Then translate back to model parameters
![Page 19: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/19.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 19
Frequency method: the big problem
Observe 5 events. Expected background of 0.9 events.Data = signal + background
Say with 68% confidence: data in range 2.84 to 8.38So say with 68% confidence: signal in range 1.94 to
7.48Suppose expected* background 4.9. Or 6.9. Or 10.9 ?“We say at 68% confidence that the number of signal
events lies between -8.06 and -2.52”This is technically correct. We are allowed to be wrong
32% of the time. But stupid. We know that the background happens to have a downward fluctuation but have no way of incorporating that knowledge
*We assume that the background is calculated correctly
![Page 20: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/20.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 20
Strategy 1: Bayes
Prior is uniform for positive , zero for negative . No problem.
Get requirement (for n observed, known background b, 90% upper limit)
0.1=nexp(-+-b) (++b)r/r!
nexp(-b) br/r!Known as “the old PDG formula” or
“Helene’s formula” or “that heap of crap”
![Page 21: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/21.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 21
Strategy 2: Feldman-Cousins
Also called* ‘the Unified Approach’Real physicists wait to see their result and then
decide whether to quote an upper limit or a range.This ‘flip-flopping’ invalidates the method.They provide a procedure that incorporates it
automatically, and always gives non-stupid results.Critics say (1) can lead to experiments quoting a
range when they’re not claiming a discovery (2) is computationally intensive and (3) For zero observed events, the higher the background estimate the better (i.e. lower) the limit on signal
* By Feldman and Cousins, principally
![Page 22: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/22.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 22
Strategy 3: CLs
As used by LEP Higgs working groupGeneralisation of Helene formulaSome quantity Q. Could be number of
events, or something more cleverCLb=P(Q or less|b) CLs+b=P(Q or less|s+b)
CLs=CLs+b/CLb
Used as confidence level. Optimise strategy using it and quote results
![Page 23: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/23.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 23
2(+) parameters Fix b, find 68% confidence
range for a, using ln L=-½
Fix a, find 68% range for bCombination (square) has
0.682=46%
a
b
L(a,b)
ln L=-½ circle has 39% ConfidenceDefine regions through contours of log L – Confidence
content given by 2ln L= for which P(n)=CL Caution! Cannot redefine a as b+c+d, claim 3
parameters and cut with P(3) instead of P(1)
![Page 24: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/24.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 24
SummaryPrediction
confronts data & sees
small/zero signal
Frequentist
probability and
Confidence
Level languageBayesian
Probability
(Health
Warning)Gaussian
ln L= -½
Zero events
Few events:
Confidence belt The horrendous
case of large
backgrounds
Extension to
several
parameters
Likelihood
![Page 25: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/25.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 25
Remember!
Zero events = 95% CL upper limit of 3 events
If it’s more involved, plot the likelihood function and use ln L=-½ for 68% central, etc
Be suspicious of anything you don’t understand
If you’re integrating the likelihood you are a Bayesian. I hope you know what you’re doing.
![Page 26: Everything You Always Wanted To Know About Limits*](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813d08550346895da6b151/html5/thumbnails/26.jpg)
Roger Barlow: YETI06 Everything you wanted to know about limits
Slide 26
Further Reading
• Workshop on Confidence Limits, CERN yellow report 2000-005
• Proc. Conf. Advanced Statistical Techniques in Particle Physics, Durham, IPPP02/39
• Proc. PHYSTAT03 – SLAC-R-703• Proc PHYSTAT05, Oxford -
forthcoming