why do wouter (and atlas) put asymmetric errors on data points ? what is involved in the cls...
TRANSCRIPT
Why do Wouter (and ATLAS) put asymmetric errors on data points ?
What is involved in the CLs exclusion method and what do the colours/lines mean ?
ATLAS J/Ψ peak (muons)Excluding SM Higgs masses
LEP exclusion Tevatron exclusion
Why do you put an error on a data-point anyway ?
ATLAS J/Ψ peak (muons)
Estimate of underlying truth (model value)
Poisson distribution
Poisson distribution
Probability to observe n eventswhen λ are expected
λ=4.90
Number of observed events
#observed #observed Lambda hypothesisLambda hypothesis
fixedfixedvaryingvarying
€
P(n | λ ) =λne−λ
n!
€
P(0 | 4.9) = 0.00745
€
P(2 | 4.9) = 0.08940
€
P(3 | 4.9) = 0.14601
€
P(4 | 4.9) = 0.17887
Poisson distribution: properties
Poisson distribution
properties
the famous √N
(1) Mean:
(2) Variance:
(3) Most likely value: first integer ≤ λ
http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf
€
P(n | λ ) =λne−λ
n!
€
⟨x⟩= λ
€
⟨(x − ⟨x⟩)2⟩= λ
Large number of events
λ=40.0
Unfortunately this is not what you wanted to know …
What you have: What you want:
€
P(Nobs | λ )
€
P(λ |Nobs)
From data to theory
Likelihood: Poisson distribution“what can I say about the measurement (Number of observed events) given an expectation from an underlying theory ?”
This is what you want to know: “what can I say about the underlying theory given my observation of a given number of events ?”
€
P(λ |Nobs) = P(Nobs | λ )P(λ )
λ (hypothesis)
Nobs known (4) information on lambda
“Given a number of observed events (4): what is the most likely / average / mean underlying true vanue of λ ?”
#observed #observed Lambda hypothesisLambda hypothesis
fixedfixed
P(N
ob
s=
4|λ
)
varyingvarying
Normally you plot -2log(Likelihood)
Likelihood:
€
P(4 | 0) = 0.00000
€
P(4 | 2) = 0.09022
€
P(4 | 4) = 0.19537
€
P(4 | 6) = 0.13385
Properties of P(λ|N) for flat P(λ)
properties
(1) Mean:
(2) Variance:
(3) Most likely value: λmost likely = x
http://www.nikhef.nl/~ivov/Statistics/Poisson.pdf
€
P(λ |Nobs) = P(Nobs | λ )P(λ )
€
⟨λ⟩ =x +1
€
⟨(λ − ⟨λ ⟩)2⟩= x +1
Assuming P(λ) is flat
This is normally presented as likelihood curve
λ (hypothesis)
P(N
ob
s=
4|λ
)-2
Log
(P(N
ob
s=
4|λ
))
Likelihood
-2Log(Prob)
4.002.32
-1.68
6.35
+2.35
sigma: ΔL=+1 ΔL=+1
68.4%
Pdf for λ
ATLAS J/Ψ peak (muons)
€
4−1.68+2.35
So, if you have observed 4 eventsyour best estimate for λ is … :
CLS method
http://www.nikhef.nl/~ivov/Statistics/thesis_I_v_Vulpen.pdf
Chapter 7.4
Your Higgs analysis
Discriminant variable Discriminant variable
Higgs
SM
Hebben we nou de Higgs gezien of niet ?
Higgs
SM
SM+Higgs
Scaled to correct cross-sections and 100 pb-1
Can also be an invariant mass plot
Approach 1: counting
Discriminant variable Discriminant variable
tellen tellen
Experiment 1 Experiment 2
Origin # events
SM 12.2
Higgs 5.1
MC total 17.3
Data 11
Origin # events
SM 12.2
Higgs 5.1
MC total 17.3
Data 17
Expectations
If the Higgs is NOT there:On average 12.2 events
If the Higgs is there:On average 17.2 events
Experiment 2:17 events observed
Experiment 1:11 events observed
SM SM + Higgs
Discovery
- Only look at what you expect from Standard Model background- Given the SM expectation: if probability to observe as many events you have observed (or more) is smaller than 5.7 10-7
SM hypothesis is very unlikely reject SM discovery !
- Only look at what you expect from Standard Model background- Given the SM expectation: if probability to observe as many events you have observed (or more) is smaller than 5.7 10-7
SM hypothesis is very unlikely reject SM discovery !€
Ppoisson (N |NSM )dN < 5.7Nobs
∞
∫ 10−7
Test hypotheses: rules for discovery
In the hypothesis that there is NO Higgs (SM hypothesis):
What is the probability to observe as many events as I have observed …OR EVEN MORE
If P < 5.7 10-7 reject SM
P(N≥33|12.2) = 6.35 10-7
P(N≥34|12.2) = 2.24 10-7
P(N≥33|12.2) = 6.35 10-7
P(N≥34|12.2) = 2.24 10-7
Integrate this plot
SM + HiggsSM
Question 1: did you make a discovery ?
See previous slide:
Yes
Discovery No discovery
No€
Ppoisson (N |NSM )dN < 5.7Nobs
∞
∫ 10−7€
Ppoisson (11 (or17) |12.2)dN > 5.711 (or 17)
∞
∫ 10−7
Question 2: did you expect to make a discovery:
If the Higgs is NOT there:On average 12.2 events
If the Higgs is there:On average 17.2 events
If you observe exactly the number of events you expect (assuming the Higgs is there), it is not unlikely enough to be explained by the SM
NO discovery expected
If you observe exactly the number of events you expect (assuming the Higgs is there), it is not unlikely enough to be explained by the SM
NO discovery expected
SM SM + Higgs
€
Ppoisson (N |12.2)dN = 0.0717
∞
∫
Question 3: At what luminosity do you expect to make a discovery ?
Lumi x 1Lumi x 1
NSM = 12.2
NHiggs = 5.1
NSM = 122.0
NHiggs = 51.0
Lumi x 10Lumi x 10
Lumi x 12.5Lumi x 12.5
NSM = 152.5
NHiggs = 63.75
no
no
yes
SM + Higgs
SM + HiggsSM
SM
€
Ppoisson (N |12.2)dN = 0.0717
∞
∫
€
Ppoisson (N |122)dN = 5.5 10−6
173
∞
∫
€
Ppoisson (N |152.5)dN = 5.2 10−7
216
∞
∫
Discovery or not
It is not likely you get exactly the number of events you expect.
You can be lucky … or unlucky.
From simple counting to the real thing in 3 steps
1) Introduce X (Likelihood ratio) test statistic
2) From simple counting to weighted counting (a real analysis)
3) Toy Monte-Carlo (fake experiments)
From simple counting to the real thing in 3 steps
1) Introduce X (Likelihood ratio) test statistic
2) From simple counting to weighted counting (a real analysis)
3) Toy Monte-Carlo (fake experiments)
Hypothesis testing: likelihood ratio
frequently used: X=-2ln(Q)
Hypothesis 1: the Standard Model without the Higgs boson
Hypothesis 2: the Standard Model with the Higgs boson
Definieer een statistic (= variabele) die onderscheid maakt tussen de 2 hypotheses.Note: kan vanalles zijn: # events of Neural net output.
Ex: counting experiment
€
Q =Ls+b
Lb
€
Q =Ppoisson (n | λ s+b)
Ppoisson (n | λ b)
Likelihood ratio
Likelihood ratio: counting
Counting experiment
N events left after some a selection of cut on discriminant
Note: X = 0 means hypoteses equally likely
Used in plots:
More SM+Higgs like More SM like
100.000 SM experiments
100.000 SM + Higgs experiments
€
Q
€
Q =P(N | s +b)
P(N | b)
€
=e−(s+b)(s +b)n /n!
e−bbn /n!
€
=e−s (s +b)n
e−bbn
€
X = −2ln(Q)
14 events observed
Variabele transformatie
Likelihood ratio: counting
Counting experiment
N events left after some a selection of cut on discriminant
Note: X = 0 means hypoteses equally likely
Used in plots:
More SM+Higgs like More SM like
100.000 SM experiments
100.000 SM + Higgs experiments
€
Q =P(N | s +b)
P(N | b)
€
=e−(s+b)(s +b)n /n!
e−bbn /n!
€
=e−s (s +b)n
e−bbn
€
X = −2ln(Q)
€
P(14 |12.2) = 0.093
€
P(14 |17.3) = 0.076
€
X = 0.420
14 events observed
€
P(15 |12.2) = 0.076
€
P(15 |17.3) = 0.087
€
X = −0.278
15 events observed
From simple counting to the real thing in 3 steps
1) Introduce X (Likelihood ratio) test statistic
2) From simple counting to weighted counting (a real analysis)
3) Toy Monte-Carlo (fake experiments)
Likelihood ratio
Counting experiment Weighted counting experiment
Eveny event has a weight according to a NN output or discriminant called pi : Signal: S(pi) and Background B(pi)
B(pi)
S(pi)+B(pi)
N events left after some a selection of cut on discriminant
tellen
€
Q =e−(s+b)(s +b)n /n!
e−bbn /n!
€
Q =e−(s+b)(s +b)n /n!
e−bbn /n!⋅
sS(pi) +bB(pi)
s+bi=1
n
∏B(pi)i=1
n
∏
From simple counting to the real thing in 3 steps
1) Introduce X (Likelihood ratio) test statistic
2) From simple counting to weighted counting (a real analysis)
3) Toy Monte-Carlo (fake experiments)
Many possible experiments
Discriminant variable Discriminant variable
tellen tellen
Experiment 1 Experiment 2
1) Experiment condensed in 1 variable Note: Each experiment (read ATLAS) yields only ONE value of Q see 2 slides ago for counting example 2) Do Toy-MC experiments to study distribution of Q Note: Two distributions: for SM and SM+Higgs hypothesis
Toy Monte Carlo experiment
SM toy experiment: Draw for each bin i a random number from Poisson with μ= λSM (i)
SM+Higgs toy experiment: Draw for each bin i a random number from Poisson with μ= λSM(i)+ λSM+Higgs(i)
λSM(i)+ λSM+Higgs(i)
λSM(i)
The Higgs does not exist: 100,000 toy-experiments (SM)The Higgs exists: 100,000 toy-experiments (SM+Higgs)
With 1 and 2 sigma bands for SM hypothesis
Note (again): each experiment will produce 1 (one) number in this plot
Different masses … different cross-sections
Small Higgs cross-section Large Higgs cross-section
Two hypotheses are more apart if: 1) cross-section of Higgs is larger 2) Higgs is more different from SM
Expectation for Q or -2ln (Q): toy experiments
Probability that background resultsin the numer observed or (even) more
If 1-CLb < 5.7 10-7 we can say we reject the SM hypothesis discovery !
The famous 5 sigma
€
1- CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs
∞
∫
Probability that background results in the numer observed or less€
CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs
∞
∫Clb = confidence level in the background
SMSM
SM+HiggsSM+Higgs
Do you expect to discover Higgs with at this mass ?
Average SM+Higgs experiment: 1-CLb = 2 10^-7So yes, you expect to make a discovery IF 10xSM
The one 2-sigma is not the other 2-sigma
2.X sigma discrepancy at mh ~ 97 GeV Far away form what you expect from Higgs1.X sigma away at mh = 114 GeV Exactly what you expect from Higgs
No 5 sigma discovery what Higgs hypotheses can we reject
No discovery
No 5 sigma deviation found … what now ?
Trying to say something on the hypothesis that the Higgs exists exclusion
Exclusion
€
CL s =CL s+b
CL b
< 0.05
- Look at what you expect from Standard Model +Higgs - Given the SM + Higgs expectation: if probability to observe as many events you have observed (or less) is smaller than 5% SM+Higgs hypothesis is not very likely reject SM+Higgs
- Look at what you expect from Standard Model +Higgs - Given the SM + Higgs expectation: if probability to observe as many events you have observed (or less) is smaller than 5% SM+Higgs hypothesis is not very likely reject SM+Higgs
Expectation for Q or -2ln (Q): toy experiments
If CLs < 0.05 we are allowed to rejectthe SM+Higgs at 95% confidence level
The famous 95% confidence level
Probability that signal hypothesis results in the numer observed or less
€
CL b = Pb(X ≥ Xobs) = Pb(X)dXX obs
∞
∫
Cls = confidence level in the signal
SMSM
SM+HiggsSM+Higgs
€
CL s+b = Ps+b(X ≥ Xobs) = Ps+b(X)dXX obs
∞
∫
€
CL s =CL s+b
CL b
Extra Normalisation:
This is why it is called modified frequentist
CLs mean SM-only expeciment is 0.13 > 0.05 so NO !
Question 2: did you expect to be able to exclude ?
Question 3: At what luminosity do you expect to make a discovery ?
Lumi = 1x normal lumi
CLs = 0.13 no exclusion for average SM-only experiment
Lumi = 2x normal lumi
CLs = 0.034 exclusion for average SM-only experiment
#SM = 100 #H = 10
#SM = 200 #H = 20
A scan:
Luminosity / nominal luminosity
CLs
CLs = 0.05
CLs = 0.13
CLs = 0.66
CLs = 0.046
2 sigma up
1 sigma down
Si: If you would have a 1 sigma downward fluctuation, i.e. you see less events than you expect there is less room for a SM+Higgs hypothesis. In this case you would have been able to exclude it.
You expect to be able to exclude at Lumi / Lumi nominal = 1.70
Question 4: At what Higgs xs do you expect to make a discovery ?
Higgs XS = 1x normal Higgs XS
CLs = 0.13 no exclusion for average SM-only experiment
Higgs XS = 2x normal Higgs XS
CLs = 0.006 exclusion for average SM-only experiment
#SM = 100 #H = 10
#SM = 100 #H = 20
A scan:
Higgs XS / nominal Higgs XS
CLs
CLs = 0.05
CLs = 0.13
CLs = 0.66
CLs = 0.046
2 sigma up
1 sigma down
You expect to be able to exclude at Higgs XS / Higgs XS nominal = 1.40
A projection along the CLs = 0.05 line
Hig
gs
XS
/ n
om
inal H
igg
s X
S
Nominal luminosity
SM only (mean)
At what Higgs XS scale factordo you expect to be able to exclude the Higgs hypothesis ?
SM only (1 sigma up)
SM only (2 sigma up)
SM only (2 sigma down)
SM only (1 sigma down)
1.4
Hig
gs
XS
/ n
om
inal H
igg
s X
S
1.4
You can now scan over Higgs masses
The important thing is of course what you actually measured